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1 . 


INTRODUCTION 


This report summarizes the results of a multi-processor 
systems design review. The systems were reviewed against use 
in a "Space Station Environment". 

The purpose of the review was to evaluate the proposed 
designs from a systems viewpoint in general and from the systems 
software in particular. The recommendations resulting from this 
evaluation are anticipated to be considered for the design of a multi- 
processing system built around a SUMO. 

Two multi-processing system designs were reviewed; one 
design proposed by the IBM Corporation, one design as proposed by 
the RCA Corporation. It was the original intent to also review a design 
proposed by the Intermetrics Corporation. However, the Intermetrics 
design was not available in time for a full evaluation, and was there- 
fore excluded from this report. 

In general, the designs reviewed were highly functional and 
many questions could not be answered. However, several major issues 
were uncovered which could be evaluated to some detail, and which 
could greatly impact the SUMC-MP design. 

The major issues relevant to amulti-processing system design 
revolve around the following functions: 

1 ) Storage Management ' 

2) Processor Management 

3) Intermodule Communication 

4) Memory Access Interference 

5) System Efficiency 

6) System Recovery/Reliability 

Section 2 of this document summarizes the results of the IBM 
design review. Section 3 summarizes the results of the RCA design 
review. References are provided at the end of this report. 
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2 . 


IBM DESIGN EVALUATION 


Two designs were, in effect, submitted by IBM. Paragraph 
2. 1 summarizes the evaluation of the first design. Paragraph 2. 2 sum 
marizes the evaluation of a second design, an amended version of 
the first. 


The first design contained many shortcomings and was con- 
sidered the lesser of the two designs (IBM and RCA). The amended 
version improved the design considerably; however, it was far more 
functional than the RCA design and therefore difficult to compare. 
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2. 1 IBM DESIGN - VERSION 1 - REVIEW SUMMARY 

MAIN TOPICS 

y SYSTEM OVERVIEW 

/ CONFIGURATION CONTROL UNIT 
o Address translation 
o Communication control 
o State control 

/ RECONFIGURATION STATE MONITOR UNIT 
/ INPUT/OUTPUT 

/ MULTIPROCESSING INSTRUCTIONS 
y FAULT TOLERANCE 
y SUMMARY 
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SYSTEM OVERVIEW 


o Processor architecture is S360 based - but not 

compatible. 

o High emphasis on automatic partitioning into inde- 

pendent systems without physical disconnects. 

o High emphasis on not affecting SUMC design (as it 

existed at that time). 

o Floating Executive intended. 
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^CONFIGURATION CONTROL UNIT 
(CCU) 


Purpose; 

o To allow logical addressing of memory modules - ATU. 

o To allow partitioning of modules into independent 

systems, without physical disconnects. - Communica- 
tion control. 

o To allow controlled loading, initializing, reconfiguration, 

etc. , - State, control. 

Implementation Notes ; 

o Each module has a CCU. 

o Special instructions provided for CCU control. 
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• ADDRESS TRANSLATION UNIT 


Purpose ; 

o To allow physical modules to be replaced without 

affecting software addresses. 

o To assign preferential storage addresses (0-n) to 

physical locations. 

Functional Characteristics ; 

o Each CPU has an ATU. 

o Each CPU may address any module. 


o 


ATU may be used to prevent access to physical module 
(partitioning into independent systems). 



• , ADDRESS TRANSLATION UNIT 
(continued) 


Comments ; 

o Re-assignment of memory module addresses can be 

more cost effectively performed in the memory module 
(HUGHES- ARMMS). 

o Design will be dictated by selected paging mechanism. 

o However PSA translate is a distinct problem unsolved 

by paging - GPU's address same logical addresses. 



PREFERENTIAL STORAGE AREA 


Purpose ; 


Contains information necessary to perform CPU 
initialization. 

Contains save areas for interrupt handling. 


Contains I/O communication control information. 


o 


Used for diagnostic logouts. 


Basic Need ; 

o Each CPU needs to generate identical fixed logical 

addresses for various communication purposes. 


o 


Logical addresses need to be transformed to physical 
addresses unique to each CPU. 
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PREFERENTIAL STORAGE AREA 
(continued) 




S360 


o 

Initialization 




- 

Initial program load PSW 



- 

Initial program load CCWl 



- 

Initial program load CCW2 


o 

Interrupt/Trap Control 



- 

External old PSW 



- 

Supervisor call old PSW 



- 

Program check old PSW 



- 

Machine check old PSW 



- 

I/O old PSW 



- 

External new PSW 



- 

Supervisor call new PSW 



- 

Program check new PSW 



- 

Machine check new PSW 



- 

I/O new PSW 


o 

I/O Control 




- 

Channel status word 



- 

Channel address word 


o 

Timer 


it- • • 


- 

Timer 


o 

Diagnostics 




- 

Model dependent 
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COMMUNICATION CELL ADDRESSING 
SUMC/MP 

o Special Translation Unit undesirable 

- Translation unit used for paging. 

Translation unit may not be unique for each 
CPU. 

o Other approaches 

Generate unique logical (ioCo, paged) address 
internal to CPU, based on (externally) assigned 
identity. 

- Use pointer-addressed common communication 
area, put CPU identity in communication cell. 
Locking of (shared) pointer (s) may cause problems. 
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• COMMUNICATION CONTROL 


Purpose ; 

o To logically disconnect modtiles from each other to 

form independent systems for: 

Maintenance 
Checkout 
Debug ( ? ) 

Bring up new experiments (?) 

Functional Characteristics ; 

o Each module has a communication state register to 

inhibit unwanted communications. 




CPU , 

IOC 

MEM 


MEM 

15 

15 

- 

to 

CPU 

4 

- 

- 

from 

CPU 

4 

4 

4 

to 

IOC 

3 

- 

- 

from 

IOC 

3 

_ 

3 


o Registers are controlled through special instructions. 
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• COMMUNICATION CONTROL 
(continued) 


Comments ; 

o Need for debug or independent systems questionable - 

system always needs protection from software errors. 


o Seems very useful for maintenance and checkout, which 

may bypass other system protection features. 
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STATE CONTROL 


Purpose; 

o 


Comments ; 

o 


To enable controlled initialization, reconfiguration 


Necessary, but is driven by other design features 


- 14 - 


\/lNPUT/OUTPUT 
(I0M DESIGN) 



o Conventional (360) approach 

o IOC is pure channel control unit 

o I/O termination interrupts handled by specific (initiating) CPU 

o Vehicle bus controller likely to be separate, and more intelligent, 

unito 


- 15 - 












^/O Exec. 

Commiinication 
^ 


I/O Device 
Controllers 


Execution of I/O Exec and Task Exec physically separated. 
lOP is (near) full-fledged processor. 

I/O termination signals can be decoupled from specific (initiating) 
CPU. 

I/O Exec integral part of lOP design. 

Vehicle bus controller may be regular lOP. 
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reconfiguration state monitor unit 

(RSMU) 


Purpose; 

o To detect major malfunctions during reconfiguration 

Functional Characteristics ; 

o Detects multiple state 3's (reconfiguration state), 

o Detects excessive time in state 3. 

o Activates alarms and generates an interrupt to the 

CPU's. 

Implementation Characteristics ; 

o Duplexed Hardware 

Operational Characteristics ; 

o Initiated by CPU 
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RECONFIGURATION STATE MONITOR UNIT 


(RSMU) 

(continued) 


General Comments ; 

o Special emphasis on reconfiguration state not justified, 

o Performs standard "watchdog" service necessary for 

any running task (exception detection of multiple state 
3's). 

o Reconfiguration is a manually initiated and controlled 

operation. 

Recommendation ; 

o Drop further consideration of this unit. 

o Use standard external interval timer to provide watchdog 

service. 
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MULTIPROCESSING INSTRUCTIONS 

o Load/Store CCU/CPU 

CCU/E 

To allow control of configuration control registers 
and states of modules. 

o Test and Set 

To perform locking of shared instructions or data. 

o Delay (n microseconds) 

Main use in conjiinction with Test and Set. 

o Store Identity 

To be used with exec, interface. 
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FAULT TOLERANCE 


o "Standard self test software Eind hardware, augmented 

by redundant elements to aid in detection and isolation 
of faults". 

o Configuration Control Registers, 

o Reconfiguration State Monitor Unit, 

o Memory parity, 

o Storage protection - lock and key. 

o Memory module addresses re-assignable (through ATU). 

o Memory access timeouts. 
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/ SUMMARY 


Design is largely traditional 

r- 


DELAY instruction seems useful 


PSA design is not f\ally worked out in RCA design 


RCA-IOP design preferable - probably 
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2. 2 IBM - VERSION 2 - REVIEW SUMMARY 
NOTABLE DIFFERENCES FROM VERSION 1 

o INTERMODULE COMMUNICATION 

Changed from busses (4) to communication matrices. (2) 

Comment: Seems better. Not enough info in handout 
to evaluate. 

o PROCESSOR ARCHITECTURE 

Changed to "augmented" S360 
No comment 

o PREFERENTIAL STORAGE ADDRESSING 

Changed from address translate to control (base) 
register. 

Comment: Good chance. 

o VIRTUAL STORAGE 

Added to system architecture 

Comment: Use of "segments" and "reference bits" unclear. 
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IBM - VERSION 2 


NOTABLE DIFFERENCES FROM VERSION 1 
(continued) 


o VIRTUAL MACHINE 

Added to Executive Design 

Comments: Sensible addition, as long as application 
programs can run on real machine in 
parallel with virtual machine. Approach 
not clear from handout. 


RECONFIGURATION STATE MONITOR UNIT 
Removed 
Comment: Good 


o 


CONFIGURATION CONTROL UNIT 


Automatic partitioning replaced by manual partitioning. 

1 

Comment: Good 


o MEMORY MODULE FEATURES 

Content correctable (?) 

Interleave 
Memory registers 

Comment: Logical changes. Not sure above effectiveness 
of memory interleave in a paged system, but 
will not hurt. 
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IBM - VERSION 2 


SUMMAR Y 


Much improved over Version 1 


No new features evident that should be considered in 
RCA design. 


Virtual machine concept worthwhile addition to RCA 
Executive. 
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3 . 


RCA DESIGN EVALUATION 


The RCA design was documented in far more detail than 
the IBM design and thus subject to closer scrutiny. The design 
was certainly the more interesting of the two. 

It is anticipated that the SUMC Multiprocessor Syste^m will 
be functionally closer to the proposed RCA design. 
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3. 1 RCA DESIGN - REVIEW SUMMARY 


GENERAL COMMENT 

Emphasis of this evaluation is the system not the software 
complexity. 


MAIN TOPICS 


Virtual Memory 
Hardware Task Switching 
Intermodule Communications 
^ Information Protection 
y Fault Tolerance 
y Potential SUMC Design Impacts 


MAIN CRITICISMS 


CPU assignment to tasks too inflexible 
Paging impact on performance too large 


^ Emphasis on system throughput not compatible with 
real time aspects. 


- 26 - 



/ VIR TUAL MEMOR Y 


o ADVANTAGES 

- Relieves application progranlmer from 
segmenting and overlaying programs. 

Allows application program execution to be 
started, even though insufficient high speed 
storage is available to contain whole program. 

o DISADVANTAGES 

Potentially large impact on performance I ! 

- Slightly increases Control Executive/Hardware 
complexity. 

Requires "look-ahead" on Variable Field length 
and I/O instructions. 




- 27 - 



NUMBER OP TRACE TAPES IN PERPORMANCE RANGE 


VIRTUAL MEMORY 
SAMPLE PAGING PERFORMANCE 


Model 65 performance relative to $lngle*level storage operating at cache 
speed 



PERCENTAGE OP IDEAL PERFORMANCE 


Parameters 

o Speed ratio; Aux Mem/HS Mem 10 

o "Page" size: 64 bytes 

o Aux Mem path width; 16 bytes (1/4 page) 


o 4 way interleave 


o Effective transfer speed per Aux Mem word ~ HS Mem word 

access time 
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Y VIRTUAL MEMORY 

o APPLICABILITY TO SUMC/MP 

Application program > high speed store 
Not a common occurrence. 

Total application program storage > high 
speed store. 

More common due to mviltiprocessing/ multi- 
pro g r amming . 

Main advantage on SUMC/MP is potentially 
more efficient high speed store utilization. 

I 

Therefore, better throughput. 

Better throughput can only be realized if page 
faulting and paging is kept to absolute minimum 
necessary. 


Conclusion: 


Note: 
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/ VIRTUAL MEMORY 
PAGING 


o GENERAL COMMENTS 

Currently most popular approach to Virtual 
Memory implementation. 

Subject to "internal fragmentation", thus off- 
setting potential storage efficiency somewhat. 

o DESIGN PARAMETERS 

Fetch Policy: When is a page to be moved to 

high speed memory? 

Replacement Policy: Which page in high speed 

memory will be removed to auxiliary memory? 

Page Size 

o MAIN IMPLEMENTATION PROBLEM 

Page address translate mechanism 
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y VIRTUAL MEMORY 
RCA DESIGN 


o FETCH POLICY 


Demand paging, except for 
executive. 


o REPLACEMENT 


POLICY 


Locked pages. 


o PAGE SIZE 


128/256 words. Due to selection 
of aux. storage parameters. 


o OTHER 


Name space > virtual memory. 
Each task has translate table. 
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y VIRTUAL MEMORY 


RCA DESIGN 
(continued) 


High Speed Store (1 us) Aux. Mem. 



10 reserved for Exec. 

1280/2560 words directly addressable application storage 
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VIRTUAL MEMORY 


RCA DESIGN 
(continued) 


ADVANTAGES 

ATU search time only holds up one CPE. 
r Application related info, protection may be 
included in ATU - on a page basis. 

DISADVANTAGES 

ATU reload, for both CPE and lOU, is a 
cominon occurrence. 

Requires additional memory accesses even though 
page is in high speed storage. 

Page swap potentially involves multiple CPE's/IOU' 
Page size too small for disk/drum. 

Duplication of entries for Exec, pages. 




Jobs 










CAM - 8 entries 


All high speed storage directly accessible 
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VIRTUAL MEMORY 


PROPOSED APPROACH 


o ADVANTAGES 

All high speed storage directly accessible by all 
CPE's/IOU's. 

Potential mix of paged and non- paged memories 
possible. 

ATU's are maintained without CPE participation. 

o DISADVANTAGES 

ATU search holds up all CPE's/IOU's. However, 
search time short and probably overlapped with 
current memory cycle. 

Page fault indicated by time out. 
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HARDWARE TASK SWITCHING 


RCA DESIGN 


Systems 

Engineering 


Executive 


User 


ROUTINE FUNCTION 

PRIORITY LEVEL 

Power Failure 

15 


Module Failure Entry 

14 


Mode Switching 

13 


MMU Code Check Failure 

12 


Executive Entry 

11 


Task Control 

10 


I/O Request 

9 


Page Fetch 

8 


Interval Timer 

7 

•H 



o 



• r4 

User Level 6 

6 


User Level 5 

5 


User Level 4 

4 


User Level 3 

3 


User Level 2 

2 


User Level 1 

1 


User Level 0 

0 



o Each level has own set of registers, 

o Registers are never automatically stored in memory, 

o Tasks are pre-assigned to user levels. 


- 37 - 




^hardware TASK SWITCHING 

RCA DESIGN 
(continued) 

o GENERAL COMMENTS 

Performance advantage negligible - with one 
exception. 

Pre- as signing tasks to CPE's is unsound policy. 

- Not re-assigning waiting tasks to other CPU's is 
unsound policy. 

- Most common need for store/restore - subroutine 
usage - not taken care of. 

However, retain large SPM and interrupt register 
Implement such that it can be used in different ways 


- 38 - 



y HARDWARE TASK SWITCHING 
PERFORMANCE EVALUATION 

I TO INITIATE A TASK UPON OCCURRENCE OF SPECIAL EVENT, 

BY JUST ENABLING LEVEL; I. E. , REGISTERS DO NOT HAVE TO 
BE LOADED. 

o System Engineering Levels 

Occurrence highly exceptional. 

- Need only a few registers to start and 
probably execute. 

Advantage negligible, 

o Executive Levels 

Occurrence regular. 

Estimate average of 4 registers contain start 
information. 

Some advantage. 

o User Levels 

Usage not applicable. The registers, if any, 
have to be loaded from memory upon assign- 
ment to CPU. 

o No advantage. 
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y' HARDWARE TASK SWITCHING 

PERFORMANCE EVALUATION 
(continued) 

TO SAVE TASK INFORMATION UPON PRIORITY INTERRUPT, 
SUCH THAT IT CAN BE AUTOMATICALLY RESTARTED WHEN 
HIGHER LEVEL TASK IS COMPLETED. 

o System Engineering Levels 

Occurrence of one event exceptional, occurrence 
of multiple events near zero. 

No advantage. 

o Executive Levels 

- Levels cannot interrupt each other, with 
exception of intermodule communication. 
Interrupt by Systems Engineering Level 
exceptional. 

Advantage, at best, doubtful, 
o User Levels 

Suspension (necessary save of registers) is 
common occurrence. 

- Re-activation on same processor not a com- 
mon occurrence. 
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HARDWARE TASK SWITCHING 

PERFORMANCE EVALUATION 
(continued) 


1) Interrupt by SE Levels 

Re-activation on same processor unlikely. 
Occurrence exceptional. 

No advantage. 

2) Interrupt by Executive Levels 

Re -activation on same processor normal for 
calls not causing "wait" condition. 

Regular occurrence. 

Advantageous. 

\ 

3) Interrupt by User Level 

Re-activation on same processor doubtful. 
Could be regular occurrence. 

- No advantage. 
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^HARDWARE TASK SWITCHING 

PERFORMANCE EVALUATION 
(continued) 


SUMMARY 


o Only advantage on certain Executive calls. 

o Number of levels cind separate level registers can be 

decreased without affecting performance, 

o Re-assigning "suspended" tasks to other CPE incurs 

large overhead. 

o Subroutine call store /re store is not handled. 
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hardware task switching 


PROPOSED IMPROVEMENTS . 
(CPE) 


Priority Levels and Assignments 


Mandatory 
Register Sets 


R outine 
Function 


Priority 

Levels 


1 


System Engineering TBD 


1 


Cross Module Comm. 1 


1 

1 

TBD 

4-5 


Interval Timer 
Exec. Call 
Processing Trap 
User Level 


1 

1 

TBD 

None 


User level has 4-5 hierarchical levels invoked on subroutine 
call (local). 


Additional instructions 

CALL SUBROUTINE (local) . 
STORE/RESTORE REGISTERS LEVEL X 


intermodule communication 

RCA DESIGN 


FUNCTIONAL CHARACTERISTICS 
CPU to CPU: None 

CPU to lOP: Stack - Global 


lOP to CPU: 


Stack - Local 


lOP to lOP: Single - Local 


IMPLEMENTATION CHARACTERISTICS 

All communication through fixed memory locations 

- Attention signal derived from continuous module 
polling of specific memory locations. 

Polling rate indeterminable. 

Polling performed by software. 


- 44 - 








J' INTERMODULE COMMUNICATION 
REQUIREMENTS ANALYSIS 

o INPUT/OUTPUT SUPPORT - Solicited 

CPU-5* (any) lOU 

Purpose: Request for data transfer 

Type: Global Request - Any lOU 

Info: Command info. 

Data address 
Request ID (not CPU ID) 

Request Priority 
lOU — > (undetermined) CPU 
Purpose: Signal Request completed 

Type: Global - CPU assignment not known 

to I/O manager 
Info: Request ID 

Request Status 
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/ INTERMODULE COMMUNICATION 
REQUIREMENTS ANALYSIS . 

o INPUT/OUTPUT SUPPORT - Solicited 

CPU-^ (any) lOU - special case 

Purpose: Feed/request data to/from continuous 

I/O task (e. g. , telemetry) 

Type: Global - lOU assignment unknown 

Info: Command Info 

Data address 

Evaluation Notes 

1) Only CPU ID is known 

2) Request status goes to specific CPE 
consequence of 1. Requires task to 
remain with specific CPE. 
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INTERMODULE COMMUNICATION 
REQUIREMENTS ANALYSIS 

INPUT/OUTPUT SUPPORT - unsolicited 
lOU-^(any) CPU 

Purpose; To signal that (uninitiated) external event 
has occurred. 

Type: Global - CPU assignment unknown. 

Info; Command Info. 

Data address, if any. 

- Evaluation Notes 

1) Design requires signal to specific CPU. 

2) This and previous case may be solved by 
lOU executing CPE Task Control. 
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\/ INTERMODULE COMMUNICATIONS 
REQUIREMENTS ANALYSIS • 

o CPU TASK ASSIGNMENT 

CPU~?>CPU 

Purpose: To assign a (higher priority) task to an 

active CPE. 

Type: Local - Specific CPE 

Info: Task control block 

Evaluation Notes: 

1) Not available in design 

2) May be solved by lOU executing CPE task 
control, 

3) May be solved by assigning (hardware) 
Executive Control to "lowest priority" CPU, 
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INTERMODULE COMMUNICATION 


REQUIREMENTS ANALYSIS 


lOU TASK CONTROL 
lOU-^IOU 

Similar to CPU task control 
Evaluation Notes 

Single cell available for lOU— t^IOU communication 



/ INTERMODULE COMMUNICATION 
RCA DESIGN 

COMMENTS 

o lOU REQUIRED TO PERFORM 

Part of CPE task control 
Intertask synchronization 
Reconfiguration 

- lOU associated functions 
Consequences 

- No separation of I/O and CPE management 

- Potential bottleneck if one lOU required 

- lOU maybe more complex than CPE 

o POLLING ADDS INTERFERENCE 

Memory access 
Program execution 

o NOT COMPATIBLE WITH REAL TIME PROCESSES 

Rate selection critical 

o BUT HARDWARE INTERFACE IS SIMPLE 
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INTERMOQUL.E COMMUNICATIONS 


SUGGESTED IMPROVEMENTS 


FUNCTIONAL CHARACTERISTICS 


- 

CPU to CPU 

Single 

- if attention 



Stack 

- if polling 

- 

CPU to lOP 

Stack 

- Global 

- 

lOP to CPU 

Stack 

- Global 

- 

lOP to lOP 

Single 

- if attention 



Stack 

- if polling 


IMPLEMENTATION CHARACTERISTICS 

All infOo through memory 

- External attention lines 

CPU-s-CPU ] 

I Local - directed 
lOP-^IOP I 


CPU^-?-IOP 


Global 





COMMUNICATION 


) IMPROVEMENTS 
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y INFORMATION PROTECTION 
RCA DESIGN 

o CONTENTS ERROR PROTECT 

II bit ECC in storage 
Burst ECC for Bus traffic 

o ADDRESS ERROR PROTECT 

Task page translate list 
Protect key 

o SHARED ACCESS PROTECT 

Lock and Go for shared instructions 

Word status bits for shared data 

Note: Currently planned to be software function 


I 
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INFORMATION PROTECTION 


RCA DESIGN 
(continued) 


o GENERAL COMMENTS 

Usage details on AEP and SAP not fully defined. 

Software interrogation of word status bits undesir- 
able. 

AEP loses power for large pages/storage blocks. 


o SUGGESTED IMPROVEMENTS 

Expand LOCK AND GO to include shared data. 
Lock word can indicate instructions or data. 

- Consider base/bound scheme for protection. 

(Writeup attached) 

1) Independent of page size 

2) Variable size blocks 


Note: There may be a distinction between Control Exec, 

protect mechanisms and application program 
protect mechanisms. 
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y' FAUL.T TOLERANCE 
RCA DESIGN 


o FEATURES 


Single error correction 

Backup copies for critical info, (implementation 
undefined) 

Module faults limited to module in error (only 
if error detected in time) 


o GENERAL COMMENTS 

Objectives should be defined in detail - unless 
not important 

- Single instruction retry should be part of 

objectives 

MIT - Hopkins, Fault Tolerant Processor 
IBM - System 370 
HUGHES - ARMMS 
INTERMETRICS - Space Station MP 
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Y POTENTIAL DESIGN IMPACTS 


o Paging related look-ahead 


o Interface with ATU 


o Interface with memory 


o Interface with channels /interrupts 


o Attention interface 


o Byte transform unit 


o Lock and go 


o Base /bound protection 
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